Sourced from github.com/twmb/franz-go's changelog.

v1.18.1

This patch release contains a myriad of fixes for relatively minor bugs, a few improvements, and updates all dependencies. Both pkg/kadm and pkg/sr are also being released as minors in tandem with a few quality of life APIs.

Bug fixes

• Previously, if records were successfully produced but returned with an invalid offset (-1), the client would erroneously return bogus offsets to the end users. This has been fixed to return -1. (Note this was never encountered in the wild).

• Pausing topics & partitions while using PollRecords previously could result in incorrect accounting in BufferedFetchRecords and BufferedFetchBytes, permanently causing the numbers returned to be larger than reality. That is, it is possible the functions would return non-zero numbers even though nothing was buffered.

• When consuming from a follower (i.e. you were using the Rack option and your cluster is configured with follower fetching), if the follower you consumed from had a higher log start offset than the leader, and if you were trying to consume from an early offset that exists on the leader but not on the follower, the client would enter a permanent spinloop trying to list offsets against the follower. This is due to KIP-320 case 3, which mentions that clients should send a ListOffsets to the follower -- this is not the case, Kafka actually returns NotLeaderOrFollower when sending that request to the follower. Now the client clears the preferred replica and sends the next fetch request to the leader, at which point the leader can either serve the request or redirect back to a different preferred replica.

Improvements

• When finishing batches, if any records were blocked in Produce due to the client hitting the maximum number of buffered records, the client would broadcast to all waiters that a message was finished for every message finished until there were no other goroutines waiting to try to produce. When lingering is enabled, linger occurs except when the client has reached the maximum number of buffered records. Once the client is as max buffered records, the client tries to flush until more records can be buffered. If you have a few concurrent producers, they will all hang trying to buffer. As soon as one is signaled, it will grab the free spot, enter into the client as buffered, and then see the client is now again at max buffered and immediately create a batch rather than lingering. Thus, signalling one at a time would cause many small single-record batches to be created and each cause a round trip to the cluster. This would result in slow performance. Now, by finishing a batch at a time, the client opens many slots at a time for any producers waiting, and ideally they can fit into being buffered without hitting max buffered and clearing any linger state. Note that single-message batches can still cause the original behavior, but there is not much more that can be done.

• Decompression errors encountered while consuming are now returned to the end user, rather

... (truncated)

• 6ea03e3 Merge pull request #893 from twmb/changelog-v1.18.1
• 6a63914 CHANGELOG: note incoming v1.18.1
• 9e81865 Merge pull request #892 from twmb/deps
• 79debef generate: remove untrue doc line
• 64fea5c franz-go: bump deps
• 868fe79 bump-deps: bump max lang supported
• 6c955bf deps: bump pkg/ (submodules)
• f2ddd13 deps: bump plugin/
• 01758cd deps: bump exampls/
• 237dc07 Merge pull request #891 from twmb/kadm-auth-ops
• Additional commits viewable in compare view

Sourced from github.com/twmb/franz-go/pkg/kadm's changelog.

v1.15.0

This release comes 74 days (just over two months) since the last minor release. This mostly contains new features, with one relatively minor but important bug addressed (and one very minor bug fixed).

Bug fixes

• Fetch sessions now properly send forgotten topics if we forget the entire topic (not just individual partitions while other partitions are still consumed on the broker). For long-running clients where partitions move around the cluster a bunch over time, this ensures we are not sending requests with null topics / null topic IDs. See #535 for more details.
• If the client talks to an http endpoint, we now properly detect the bytes 'HTTP' and send a more relevant error message. This previously existed, but used the wrong int32 internally so 'HTTP' was not properly detected (thanks @​alistairking!).

Features

• RecordReader now supports %v{json} to parse json inputs
• RecordReader now supports %v{}, an empty no-op formatting directive
• Adds PurgeTopicsFromProducing and PurgeTopicsFromConsuming to purge either the producing or consuming half of the client, if you are producing to and consuming from the same topics in the same client.
• The new ConsiderMissingTopicDeletedAfter option allows finer grained control for how long a regexp-discovered topic can be missing on the cluster before the topic is considered deleted (and then internally purged in the client)
• Adds NewErrFetch to create a single-error Fetches, for use in end-user tests / stubs.
• The new MaxBufferedBytes option can control how many bytes are buffered when producing, an option similar to MaxBufferedRecords.
• Adds BufferedFetchBytes and BufferedProduceBytes to return the total bytes in records buffered (note this counts only keys, values, and headers).
• Adds PreTxnCommitFnContext to allow custom Metadata annotation for transactional commits.
• Adds LeaveGroupContext to control how long leaving a group can take, as well as to return any leave group error.

Relevant commits

franz-go

• 4dcfb06 feature kgo: add LeaveGroupContext
• 910e91d and 60b601a feature kgo: add PreTxnCommitFnContext
• c80d6f4 feature kgo: add Buffered{Fetch,Produce}Bytes
• 304559f feature kgo: support MaxBufferedBytes

... (truncated)

• 42172c9 Merge pull request #574 from twmb/changelog-v1.15.0
• 003ffb7 changelog: update for incoming v1.15.0
• 2f8a7c4 Merge pull request #575 from twmb/precommittxn
• 60b601a kgo / changelog: last minute rename from CommitTxn to TxnCommit
• 036f599 Merge pull request #571 from twmb/kadm-lag
• 33a0fac Merge pull request #573 from twmb/misc
• e2f01a2 Merge pull request #572 from twmb/kgo-source-log
• 2655e1f kadm: document missing topics being elided when listing topics
• f31eea7 kmsg: document more errors for describe user scram credentials
• a3e5141 kgo: improve a log line
• Additional commits viewable in compare view

Sourced from github.com/twmb/franz-go/pkg/kmsg's changelog.

v1.9.0

This release contains one important bugfix (sequence number int32 overflow) for long-lived producers, one minor bugfix that allows this client to work on 32 bit systems, and a few other small improvements.

This project now has integration tests ran on every PR (and it is now forbidden to push directly to master). These integration tests run against Kraft (Kafka + Raft), which itself seems to not be 100% polished. A good amount of investigation went into hardening the client internals to not fail when Kraft is going sideways.

This release also improves behavior when a consumer group leader using an instance ID restarts and changes the topics it wants to consume from. See the KIP-814 commit for more details.

It is now easier to setup a TLS dialer with a custom dial timeout, it is easier to detect if requests are failing due to missing SASL, and it is now possible to print attributes with RecordFormatter.

Lastly, the corresponding kadm v1.3.0 release adds new LeaveGroup admin APIs.

franz-go

• b18341d kgo: work around KIP-814 limitations
• 6cac810 kversions: bump Stable from 3.0 to 3.3
• [PR #227](twmb/franz-go#227) bugfix further sequence number overflows fix (thanks @​ladislavmacoun!)
• [PR #223](twmb/franz-go#223) add GitHub actions integration test (thanks @​mihaitodor!) and [PR #224](twmb/franz-go#224) fixup kgo guts to work around new kraft failures
• 203a837 franz-go: fix 32 bit alignment, fix a few lints
• 719c6f4 kgo: avoid overflow on 32 bit systems
• db5c159 feature kgo: add DialTimeout function, complementing DialTLSConfig
• b4aebf4 kgo: add ErrFirstReadEOF, which unwraps to io.EOF
• bbac68b RecordFormatter: support %a; formatter&reader: support 'bool'

kadm

• d3ee144 kadm: add LeaveGroup api
• 7b8d404 kadm: ListOffsetsAfterMill(future) should return end offsets

• 24d0cfe Merge pull request #231 from twmb/v1.9
• b0bc0b6 CHANGELOG: note incoming v1.9, incoming kadm
• a4a2aaf Merge pull request #230 from twmb/kadm
• d3ee144 kadm: add LeaveGroup api
• 2ee43e3 kadm: update CalculateGroupLag documentation
• 00209b4 kgo: add addr to sasl logs
• a7f5d0d Merge pull request #229 from twmb/instance_id
• f8038de kgo: consistently use \d.\d rather than \d.\d.0 for kafka versions
• b18341d kgo: work around KIP-814 limitations
• 6cac810 kversions: bump Stable from 3.0 to 3.3
• Additional commits viewable in compare view

You can trigger Dependabot actions by commenting on this PR:

• @dependabot rebase will rebase this PR
• @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
• @dependabot merge will merge this PR after your CI passes on it
• @dependabot squash and merge will squash and merge this PR after your CI passes on it
• @dependabot cancel merge will cancel a previously requested merge and block automerging
• @dependabot reopen will reopen this PR if it is closed
• @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
• @dependabot show <dependency name> ignore conditions will show all of the ignore conditions of the specified dependency
• @dependabot ignore <dependency name> major version will close this group update PR and stop Dependabot creating any more for the specific dependency's major version (unless you unignore this specific dependency's major version or upgrade to it yourself)
• @dependabot ignore <dependency name> minor version will close this group update PR and stop Dependabot creating any more for the specific dependency's minor version (unless you unignore this specific dependency's minor version or upgrade to it yourself)
• @dependabot ignore <dependency name> will close this group update PR and stop Dependabot creating any more for the specific dependency (unless you unignore this specific dependency or upgrade to it yourself)
• @dependabot unignore <dependency name> will remove all of the ignore conditions of the specified dependency
• @dependabot unignore <dependency name> <ignore condition> will remove the ignore condition of the specified dependency and ignore conditions